fix(bin): detect and repair crewmate panes that survive a restart unable to work - #1421
Open
Moshik21 wants to merge 4 commits into
Open
fix(bin): detect and repair crewmate panes that survive a restart unable to work#1421Moshik21 wants to merge 4 commits into
Moshik21 wants to merge 4 commits into
Conversation
A session-provider restart can bring a crewmate pane back looking perfectly healthy while it is silently unable to work, and endpoint liveness reports it ALIVE. This happened twice on 2026-07-31 and stopped the fleet both times. Mechanism, verified against herdr 0.7.5 and Claude Code 2.1.220: herdr persists only each pane's CREATION cwd, and its default `[session] resume_agents_on_restore` relaunches the agent from the integration-reported session ref (`claude --resume <id>`). That command is herdr's own, so it carries none of the launch shape fm-spawn resolved. The pane returns without its autonomy flag - stalling forever on its first tool call - and in the project checkout, because the task worktree was only ever entered by a `treehouse get` subshell the persisted creation cwd never followed. Detection: fm-crew-fitness.sh reports whether a recorded task's pane still carries the autonomy flags its own recorded launch command specified and is still in its recorded worktree. The session-start digest runs it for every live endpoint - the crash that causes this takes the watcher down too, so a session start is where it surfaces. The flag set is intersected with the task's recorded launch rather than mapped per harness, so a template change can never leave the check asserting a stale flag. Exact argv-element matching is required, not a substring scan: a brief travels in argv and can quote the very flag being looked for. Repair: fm-spawn.sh now records the fully resolved launch command as `launch=` in the task's metadata, and fm-crew-relaunch.sh replays it in the recorded worktree. It refuses to relaunch over a live agent, refuses when the recorded worktree is unaccounted for, never allocates a second worktree, and never touches worktree contents. Both fail closed toward "unknown": a false "fit" is the exact silent failure being fixed. The endpoint is proven present through the recovery-grade state contract first, because tmux silently resolves an absent target to the caller's own window - which would otherwise condemn a vanished task by measuring firstmate's own pane. Also corrects the recorded reason a keystroke cannot repair this: bypassPermissions is a launch-time grant, so a process started without it does not offer that mode at all, independent of key transmission.
…h tool, backend limits
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Firstmate crewmate panes that survive a host crash or reboot come back looking perfectly healthy while being silently unable to work, and the existing endpoint-liveness check reports them ALIVE. This happened twice on 2026-07-31 and stopped the fleet both times. The captain asked to make firstmate detect and repair it, and required diagnosing the real mechanism first rather than building on their reconstruction.
Verified mechanism (against herdr 0.7.5 and Claude Code 2.1.220): herdr persists only each pane's CREATION cwd, and its default [session] resume_agents_on_restore relaunches the agent from the integration-reported session ref (claude --resume ). That command is herdr's own, so it carries none of the launch shape fm-spawn resolved. The pane therefore returns without its autonomy flag - stalling forever on its first tool call waiting for an approval nobody is watching for - and in the project checkout, because the task worktree was only ever entered by a treehouse get subshell whose cwd the persisted creation cwd never followed.
Detection: new bin/fm-crew-fitness.sh reports whether a recorded task's pane still carries the autonomy flags its own recorded launch command specified and is still in its recorded worktree. The session-start digest runs it for every live endpoint, because the crash that causes this takes the watcher down too, so session start is where it actually surfaces. Deliberate decisions: the flag set is intersected with the task's own recorded launch rather than mapped per harness, so a template change can never leave the check asserting a stale flag; exact argv-element matching is required rather than a substring scan, because a brief travels in argv and can quote the very flag being looked for; and both scripts fail closed toward 'unknown' rather than 'fit', since a false 'fit' is the exact silent failure being fixed. The endpoint is proven present through the recovery-grade state contract first, because tmux silently resolves an absent target to the caller's own window, which would otherwise condemn a vanished task by measuring firstmate's own pane.
Repair: fm-spawn.sh now records the fully resolved launch command as launch= in the task's metadata so recovery is deterministic and does not re-derive flags under pressure, and new bin/fm-crew-relaunch.sh replays it in the recorded worktree. It refuses to relaunch over a live agent, refuses when the recorded worktree is unaccounted for, never allocates a second worktree, and never touches worktree contents - the three safety properties the captain required be preserved.
The change also corrects a previously recorded learning about why a keystroke cannot repair this: bypassPermissions is a launch-time grant, so a process started without it does not offer that mode at all, independent of any key-transmission question about shift+tab.
Scope was deliberately held to a recovery correctness fix rather than a new supervision subsystem, per the captain's instruction. Firstmate additionally asked me to consider folding in a spawn-time task-vs-task worktree ownership assertion if it were small. I implemented it, then testing against real treehouse and herdr disproved its premise, so I deliberately did NOT ship it and am reporting it back instead - it needs a liveness contract that is larger than this branch should carry.
Covered by tests/fm-crew-fitness.test.sh (44 assertions) proving detection of both the manual-mode case and the wrong-cwd case, and proving the repair restores an autonomous agent in the correct worktree without duplicating a task or losing uncommitted work. This is firstmate shared tracked material, so it follows firstmate-coding-guidelines: one sentence per line in docs, shellcheck-clean scripts via bin/fm-lint.sh, colocated tests, no agent co-author.
What Changed
bin/fm-crew-fitness.sh, which reports whether a live task pane still carries the autonomy grants its own recordedlaunch=specified and is still in its recorded worktree, on a three-statefit|unfit|unknownverdict that fails closed towardunknown;bin/fm-session-start.shnow emits one fitness line per endpoint that read as alive, since the crash that causes this failure takes the watcher down too.bin/fm-crew-relaunch.sh, which replays a task's recorded launch command in its recorded worktree — refusing over a live or vanished endpoint, never allocating a second worktree, never touching worktree contents, and appending a recovery note to the brief first. To make that deterministic,bin/fm-spawn.shrecords the fully resolved launch command aslaunch=instate/<id>.metaand rejects newline-bearing launch strings that would corrupt the metadata.fm_backend_pane_argv_has,fm_backend_pane_env_has,fm_backend_current_path_passive, and dispatchers forsend_text_line/current_path, implemented passively for tmux (/procargv and environ) and herdr (pane process-info) and returning undeterminable elsewhere; covered bytests/fm-crew-fitness.test.sh,tests/fm-backend-herdr.test.sh, and a newtests/fm-spawn-dispatch-profile.test.shcase pinning the recordedlaunch=line, with the backend limits recorded indocs/verification/runtime-backends.md.Risk Assessment
✅ Low: Every finding from both prior rounds is fixed with behavior-level test coverage and honest verification evidence, the new env reader preserves the never-toward-fit invariant by construction, and the two trickiest new mechanisms (the array-expansion guard and the shell quote round-trip) verify correct under direct exercise.
Testing
I ran the change's own suite (tests/fm-crew-fitness.test.sh, 66 assertions) plus the adjacent herdr/backend/spawn suites, then demonstrated the intent the way the captain would actually meet it: a real tmux pane running a stand-in harness in a real task worktree with uncommitted work, restored herdr-style asclaude --resume <id>in the project checkout. The session-start digest reports that paneendpoint: aliveand immediately below itfitness: unfit (autonomy=lost cwd=wrong)naming the lost--dangerously-skip-permissionsand the wrong directory; relaunch refuses while the agent is live, dry-run changes nothing, and after exiting the agent the repair brings back an autonomous agent in the recorded worktree (fitness: fit) with the uncommitted file intact, the same worktree count, GOTMPDIR recreated, and a recovery note appended to the brief. I also found and closed a coverage gap — nothing asserted that fm-spawn.sh writeslaunch=into task metadata, the record both recovery scripts depend on — by adding a focused test that pins the recorded line to the literal command typed into the pane; it fails at the base commit and passes here. This is a terminal CLI surface with no GUI, so the reviewer-visible evidence is the captured tmux pane render and the session-start digest transcript rather than a screenshot. One adjacent suite, tests/fm-session-start.test.sh, fails on this host for a pre-existing fixture reason (real /usr/bin/node in the fixture PATH) and fails identically at the base commit.Evidence: Restart-recovery end-to-end transcript (session-start digest before/after the crash and after the repair, plus the rendered tmux pane)
Evidence: The digest lines a captain sees, before the crash / after the herdr restore / after the repair
Evidence: Reproducer for the end-to-end demo (bash demo-restart-recovery.sh <repo-root> <out-dir>)
Evidence: fm-crew-fitness suite output (detection + repair assertions)
Evidence: Spawn suite output including the added launch= recording test
/tmp/no-mistakes-evidence/01KYWSQ51EXYW8YPM3TQA704SG/adjacent-suites.log) - Outcome:Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
bin/fm-crew-relaunch.sh:139- bin/fm-backend.sh:730 documents fm_backend_send_text_line's 4th argument as [session], but bin/backends/zellij.sh:470 and bin/backends/cmux.sh:509 take it as [expected-label] (the task label fm-<id>), which their target_ready uses to verify the tab/workspace title. fm-spawn.sh passes $W (the label) so it stays correct, but bin/fm-crew-relaunch.sh:139 and :142 pass $WINDOW, the recorded window= target string (e.g. "<zellij-session>:<pane-id>"). fm_backend_zellij_target_ready then compares that against the tab label, fails, and the send returns 1 — so on zellij and cmux the relaunch always dies at line 140 with "could not send the directory change", after having already appended the recovery note to the brief. Fix: pass "fm-$ID" as the label from both new scripts and rename the dispatcher parameter to [expected-label].bin/fm-crew-fitness.sh:210- bin/fm-crew-fitness.sh:210 calls fm_backend_current_path with $window as the third argument, which zellij (bin/backends/zellij.sh:403) and cmux (bin/backends/cmux.sh:448) treat as expected-label, so target_ready fails and the cwd axis silently reads unknown on every zellij/cmux task. Worse, those two readers are not passive: they sendprintf marker; pwd; printf markerinto the pane and capture it, and their headers state they are deliberately scoped to fm-spawn's pre-launch worktree poll where injecting a command is acceptable. Correcting the label alone would therefore make fitness submit a bogus prompt into every live crewmate's composer at every session start, contradicting the script's own "Read-only and side-effect free" contract at bin/fm-crew-fitness.sh:218. Fix at the shared boundary: read the cwd axis only from backends with a passive foreground-cwd reader (tmux, herdr) and report cwd=unknown for the rest.bin/fm-crew-relaunch.sh:104- bin/fm-crew-relaunch.sh:104 licenses a relaunch when fm_backend_agent_state reportsmissing, butmissingmeans the recorded endpoint is authoritatively absent — there is no pane to send to. I verifiedtmux send-keys -t <session>:<absent-window>exits 1 with "can't find window" (no misdelivery), and herdr's send_text_line fails its target_ready, so the run always dies at line 140 with "could not send the directory change" — after the recovery note has already been appended to the brief at line 130. The header and stuck-crewmate-recovery both advertisemissingas a supported repair state, which sends an operator down a path that cannot succeed. Either refusemissingwith an explicit "the endpoint is gone; this is a respawn, not a relaunch" message, or recreate the endpoint before sending.bin/fm-crew-relaunch.sh:142- fm-spawn.sh:1687 unconditionally sendsexport GOTMPDIR=$TASK_TMP/gotmpinto the pane shell before every launch, so the agent and all its children inherit it. bin/fm-crew-relaunch.sh replays only launch=, and after a host restart the pane shell is fresh, so the relaunched agent runs without GOTMPDIR while the script reports a successful relaunch. The meta already records tasktmp=, so the fix is to send the same export line before the launch command, mirroring fm-spawn.bin/fm-crew-relaunch.sh:139- bin/fm-crew-relaunch.sh:139 sendscd $WORKTREEwith the path interpolated unquoted into the shell command line. A recorded worktree path containing a space or a glob character makes the pane runcdon a truncated path; the send itself still succeeds, so the launch command is then typed in the wrong directory and the script prints "relaunched ..." — reproducing exactly the wrong-worktree failure it exists to repair. Shell-quote the path (printf %q) in the sent command.bin/fm-crew-fitness.sh:189- The intent requires that "both scripts fail closed toward 'unknown' rather than 'fit', since a false 'fit' is the exact silent failure being fixed", and the header at bin/fm-crew-fitness.sh:212 repeats "never toward fit". But at line 189 a recorded launch whose autonomy grant is an env prefix reports autonomy=n/a, which the verdict rules treat as satisfied. That is reachable as a falsefit: opencode's grant isOPENCODE_CONFIG_CONTENT={"permission":{"*":"allow"}}(fm-spawn.sh:474), and a herdr resume drops env prefixes exactly as it drops flags. For any secondmate the recorded worktree equals the pane's creation cwd (fm-spawn.sh:806, WT="$PROJ_ABS"), so cwd reads ok after a restart — an opencode or pi secondmate therefore reportsfitness: fitwhile it has lost both its permission grant and its FM_HOME/FM_*_OVERRIDE prefixes. Consider reportingunknownrather thann/awhen the recorded launch carries a recognized env-prefix grant.bin/fm-spawn.sh:1685- bin/fm-spawn.sh:1685 appends launch= verbatim. The comment asserts "One line, no newlines: every launch template is a single command line", which holds for the templates but not for the raw-launch-command escape hatch at fm-spawn.sh:503 (LAUNCH=$ARG3, any caller-supplied string containing a space). A newline in that string writes extra lines into state/<id>.meta, and because fm_meta_get (bin/fm-backend.sh:341) takestail -1, an injected trailingworktree=orwindow=line wins over the genuine one for every consumer, including teardown. Reject or strip newlines before recording.bin/fm-backend.sh:924- fm_backend_pane_argv_has declares "1 - confidently absent" as a contract-level verdict, but its tmux reader resolves the pane throughtmux display-message -p -t <target> '#{pane_tty}', which I confirmed silently succeeds (rc=0) against an absent target and answers about the caller's own pane. The membership guard that makes the verdict trustworthy lives only in the one caller (fm-crew-fitness.sh:161's fm_backend_agent_state gate), not at the shared boundary where the contract is stated. No second caller exists today, so nothing is broken now; worth either folding the exact-window check into the primitive or stating the caller obligation in the contract comment before the next caller appears.bin/backends/herdr.sh:1903- fm_backend_herdr_pane_argv_has depends on a herdr CLI surface (pane process-infoand the.result.process_info.foreground_processes[].argvschema) that appears nowhere else in the repo, has no entry in docs/verification/runtime-backends.md alongside the other verified herdr commands, and is not exercised by tests/fm-crew-fitness.test.sh (which skips without tmux and only drives the tmux adapter). herdr is the backend the two production incidents happened on, so this is the path most likely to be exercised first. Per firstmate-coding-guidelines ("update the corresponding verification evidence when behavior changes"), record the command, version, and exact output backing the argv-array claim.🔧 Fix: harden restart repair sends and fail-closed fitness verdicts
2 issues (1 warning, 1 info) still open:
bin/fm-crew-relaunch.sh:140- bin/fm-crew-relaunch.sh:140 replaysexport GOTMPDIR=<tasktmp>/gotmpbut never creates the directory. fm-spawn.sh:1349-1355 is explicit that "Go won't create GOTMPDIR, so mkdir before it is used", and TASK_TMP is/tmp/fm-<id>— which the host reboot this script exists to recover from typically clears (systemd-tmpfiles or a tmpfs /tmp). Concrete failure: crewmate reboots, operator runs fm-crew-relaunch.sh, the agent comes back with GOTMPDIR pointing at a path that no longer exists, and its firstgo build/go testfails with a work-dir creation error instead of falling back to the default temp — a new breakage introduced by the replay, in precisely the scenario it targets. Fix:mkdir -p "$TASKTMP/gotmp"before sending (or fold the mkdir into the sent line), mirroring fm-spawn.sh:1355.bin/fm-crew-fitness.sh:96- bin/fm-crew-fitness.sh:96 justifies the env-grant cap with "NO backend can verify: a running process's environment is not readable through any session-provider surface". That holds for herdr (its process-info schema exposes argv/argv0/cmdline/cwd/name/pid and no environment), but not for the tmux adapter as implemented: bin/backends/tmux.sh:119-131 already reads /proc/<pid>/cmdline for the same foreground pids, and /proc/<pid>/environ is readable for a same-uid process (Yama's ptrace_scope gate applies to ATTACH, not READ). The practical consequence of the blanket rule is that every secondmate (all carry the FM_HOME prefix) and every opencode task now permanently reportsfitness: unknownat every session start, healthy or not — fail-closed as instructed, but a whole task class whose verdict can never be actionable. Worth either softening the claim to name the backends it is true for, or adding a passive environ read on tmux under the same three-state contract so tmux secondmates get a real verdict.🔧 Fix: verify env-prefix grants and recreate replayed GOTMPDIR
1 info still open:
bin/backends/tmux.sh:171- Both tmux readers gate on /proc (bin/backends/tmux.sh:129 for argv, :171 for environ), so on a host with no /proc - which docs/verification/runtime-backends.md:147 now states plainly is every macOS host - the autonomy axis returns undeterminable for every tmux task and fitness can only ever reportunknownthere. The cwd axis still works via#{pane_current_path}, and herdr (the backend both incidents happened on) answers argv through its own CLI, so the detection this branch exists for does fire where it was needed. Noting the boundary only for the record: it is fail-closed, honestly documented in the verification entry rather than asserted away, and needs no change here.tests/fm-session-start.test.sh:789- Pre-existing, host-dependent failure unrelated to this change: tests/fm-session-start.test.sh::test_output_ordering_diagnostics_lead fails with "MISSING diagnostic did not appear at all". The case deletes the fakenodeto force a MISSING diagnostic, but this host has /usr/bin/node, which is inside the fixture's BASE_PATH (/usr/bin:/bin:/usr/sbin:/sbin), so bootstrap still finds node. Verified it fails identically at the base commit 9fdef64, so it is not caused by this branch; no action taken.bash bin/fm-test-run.sh tests/fm-crew-fitness.test.sh— 66 assertions covering the manual-mode case, wrong-cwd case, combined crash shape, quoted-flag argv matching, env-prefix grants, fail-closed unknown paths, and the repair (refusals plus the successful relaunch)bash bin/fm-test-run.sh tests/fm-backend-herdr.test.sh tests/fm-backend.test.sh tests/fm-spawn-dispatch-profile.test.sh tests/fm-session-start.test.sh— new herdrpane_argv_hasthree-state cases and backend dispatch; session-start failure is pre-existing (see findings)Manual end-to-end demo on a real tmux server (private socket) with a real git project + task worktree holding uncommitted work:bin/fm-session-start.shdigest before the crash, after a herdr-styleclaude --resume <id>restore from the project checkout, and after repair — captured in restart-recovery-transcript.txtbin/fm-crew-fitness.sh fix-parserandbin/fm-crew-fitness.sh --allagainst the restored pane (exit=1,autonomy=lost cwd=wrong)bin/fm-crew-relaunch.sh fix-parserover a live agent (refused),--dry-run(creates nothing), and the real repair after exiting the agent — then re-checked fitness, uncommitted work,git worktree listcount, GOTMPDIR recreation, and the brief's recovery noteNewtests/fm-spawn-dispatch-profile.test.sh::test_meta_records_the_exact_launch_command_that_was_typed— pinslaunch=in state/<id>.meta to the literal command sent to the pane, as a single line, carrying the autonomy grant fitness readsMutation check: ran the new spawn test against a scratch checkout of base commit 9fdef64 — it fails there ("spawn recorded no launch= line") and passes on the target, confirming it detects the absence of the recordingtests/fm-spawn-dispatch-profile.test.sh- This file was already modified and uncommitted in the worktree when this phase started (it is not part of the base..target diff and I did not touch it). Flagging so the outer executor knows the working tree carries a test edit outside the documented change.✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.